System and method for veryifying audible and/or visual notifications

ABSTRACT

A method of verifying audio notifications generated within a vehicle includes generating an audio notification using an audio generation path that includes at least a controller, an audio notification generator and a speaker, wherein the controller generates an audio notification request utilized by the audio notification generator to generate an audio signal, and wherein the speaker converts the audio signal into the audio notification. The method further includes monitoring sounds within the vehicle using a microphone separate from the audio generation path and analyzing the monitored sounds using an audio analyzer, wherein the audio analyzer verifies whether the audio notification was generated. Feedback is provided to the audio generation path regarding whether generation of the audio notification was verified.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation application and claims benefit under 35 U.S.C. §120 of U.S. patent application Ser. No. 16/814,200, filed Mar. 10, 2020,the entire disclosure of each of which is hereby incorporated byreference.

FIELD

This disclosure is generally directed to audible and/or visualnotifications utilized in automobiles and, more specifically to systems,devices and methods for verifying generation and/or delivery of thesenotifications.

BACKGROUND

Vehicles utilize a variety of audible and/or visual notifications tocommunicate information to the operator of a vehicle. In some cases, aplurality of different components and/or systems may be utilized togenerate the audible and/or visual notification. A fault in any one ofthe components/systems included in the notification generation path mayresult in a failure to generate the notification. In some cases, thesystems utilize a plurality of internal checks to ensure componentswithin the notification generation path are operating correctly.However, the desired result is that the operator becomes aware of thecondition communicated by the audible and/or visual notification, andthe internal checks do not verify whether the operator received thecommunication. It would be desirable to remedy these deficiencies.

SUMMARY

According to one aspect, a method of verifying audio notificationsgenerated within a vehicle is provided. The method may includegenerating an audio notification using an audio generation path, whereinthe audio generation path includes at least a controller, an audionotification generator and a speaker. The controller generates an audionotification request utilized by the audio notification generator togenerate an audio signal, and the speaker converts the audio signal intothe audio notification. The method further includes monitoring soundswithin the vehicle using a microphone separate from the audio generationpath and analyzing the monitored sounds using an audio analyzer, whereinthe audio analyzer verifies whether the audio notification wasgenerated. Feedback is provided to the audio generation path regardingwhether generation of the audio notification was verified.

According to another aspect, an audio notification system for use invehicles is provided. The audio notification system include an audionotification generation path that further includes at least acontroller, an audio notification generator, and a speaker, wherein thecontroller generates a audio notification request, the audio generatorgenerates an audio notification signal in response to the audionotification request, and the speaker converts the audio notificationsignal into an audio output provided as an audio notification to anoperator of the vehicle. In addition, the audio notification system mayinclude an external audio verification system that includes at least amicrophone and an audio analyzer, wherein the microphone monitors audiosignals within an interior cabin of the vehicle and the audio analyzeranalyzes monitored audio signals to verify whether the audionotification requested by the controller was properly generated.Verification is provided as feedback to the audio notificationgeneration path.

According to another aspect, a method of verifying audio notificationsgenerated within a vehicle is provided. The method includes receivingfeedback from an external audio monitoring system that includes amicrophone and an audio signal analyzer regarding ambient sound levelsmeasured within an interior cabin of a vehicle. The method furtherincludes generating an audio notification using an audio generation paththat includes at least a controller, an audio notification generator, anamplifier and a speaker, wherein the controller generates an audionotification request utilized by the audio notification generator togenerate an audio notification signal based on the ambient sound levelsmeasured by the external audio monitoring system, and wherein thespeaker converts the audio notification signal into the audionotification. Sounds are monitored within the vehicle using a microphoneseparate from the audio generation path and the monitored sounds areanalyzed using an audio analyzer, wherein the audio analyzer verifieswhether the audio notification was generated. Feedback is provided tothe audio generation path regarding whether generation of the audionotification has been verified.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating generation and monitoring ofaudio and/or visual notifications according to some embodiments.

FIG. 2 is a block diagram illustrating generation and monitoring ofaudio notifications according to some embodiments.

FIG. 3 is a flowchart illustrating verification of audio notificationsaccording to some embodiments.

FIG. 4 is an interior view of a vehicle cabin having an external visualnotification verification system according to some embodiments.

FIG. 5 is a block diagram illustrating generation and monitoring ofvisual notifications according to some embodiments.

FIG. 6 is a flowchart illustrating verification of visual notificationsaccording to some embodiments.

DETAILED DESCRIPTION

A typical vehicle communicates information to an operator of the vehicleutilizing a variety of audio and visual notifications. For example, anaudio notification (e.g., chime) may be generated in response to anobject detected behind the vehicle while in reverse. A plurality ofcomponents are required to generate the desired audio notification, fromthe sensors utilized to detect the object, to the controller responsiblefor receiving the alert and generating the notification request, to theamplifiers and speakers utilized to generate the audio notification. Afailure at any point along the generation path may result in a failureto generate the audio notification. In addition, in some cases the audionotification (or visual notification) may be generated but may not bereceived by the operator due to factors external to the notificationgeneration path such as excessive cabin noise or obscured visualdisplay.

According to some aspects, this disclosure is directed to an externalverification system for monitoring audio and/or visual notifications toensure notifications are generated and/or received by the operator.Unlike many fault detection systems, the external verification pathmonitors the output of the notification path—the actual audionotification and/or visual notification—using one or more microphonesand/or cameras. Hence, the verification system is external to theaudio/visual notification generation path. For example, in someembodiments the external verification system is comprised of one or moremicrophones and an audio analyzer (e.g., digital signal processor)configured to monitor sounds generated within the cabin and providefeedback verifying proper generation of the audio notification. In someembodiments the feedback includes only verification of whether the audionotification was detected. In other embodiments, other parameters may bemonitored such as volume of the audio notification, the signal-to-noiseratio (SNR) of the audio notification relative to ambient noise withinthe cabin, and/or other parameters to determine whether the audionotification was likely audible to the operator. In other embodiments,the external verification system is comprised of one or more cameras anda visual analyzer (e.g., digital signal processor) configured to monitorvisual indicators displayed to an operator and/or visual responses of anoperator to verify the generation of visual indicators. In someembodiments, the feedback may additionally include parameters associatedwith the visual notification that can be utilized to determine whether avisual notification was received by the operator.

FIG. 1 is a block diagram illustrating components utilized to generatean audio and/or visual notification for display to an operator 100. Inthe embodiment shown in FIG. 1, these components include one or moresensors 102, a controller 104, an audio output device 106 and/or visualoutput device 108. For example, the sensors 102 may include a variety ofdifferent types of sensors located at various locations within andoutside of the vehicle. These may include proximity, radar, LiDARdetectors, seat belt detectors, passenger detection sensors, temperaturesensors, maintenance schedule signals, low oil sensors, gas tanksensors, and others. Controller 104 generates notification requests inresponse to the data provided by the one or more sensors 102. This mayinclude audio and/or visual notifications, wherein the audio outputdevice 106 generates audio notifications and the visual output device108 generates visual notifications. Collectively, these componentscomprise the notification generation path and are utilized to generatean audio and/or visual notification provided to the operator 100. Itshould be understood that the notification generation path may include anumber of additional components/systems, some of which are shown in moredetail in FIGS. 2 and 5, below. The output of the notificationgeneration path is either an audio notification 110 and/or a visualnotification signal 111. For example, if the notification signal is anaudio signal 110 it may be in the form of a simple tone, a complex tone,music or parts of music, a voice recording, or any other sound ormixture of the mentioned sounds that can be used to convey informationto the operator. If the output of the notification generation path is avisual notification signal 111 it may be in the form of a visualdisplay. The visual display may be generated in a plurality oflocations, including one or more of the dash display, a heads-up displayon a dedicated display or projected on the windshield, as indicatorlights as part of or separate from the main dash display, as indicatorlights or visual displays at other locations such as the steering wheel,center console, front pillar, overhead console, or center seat console.

An external verification system monitors the audio and/or visual outputs110, 111, respectively generated by the audio output device 106 and/orvisual output device 108 and provides feedback to verify delivery of theaudio and/or visual outputs. In some embodiments the externalverification system includes an external audio verification system 114comprised of one or more microphones configured to monitor sounds withinthe cabin. The external verification system 114 provides feedback to theaudio notification generation path verifying generation of the audionotification. In some embodiments, the independent monitoring systemincludes a visual monitoring system 116 comprised of one or more camerasconfigured to monitor visual displays within the vehicle and providefeedback to the notification generation path verifying generation of thevisual notification. In some embodiments, the independent monitoringsystem is comprised of both an audio monitoring system 114 and a visualmonitoring system 116. It is important to note that the audio monitoringsystem 114 and/or visual monitoring system 116 are not a part of thenotification generation path 108. Both the audio monitoring system 114and visual monitoring system 116 are configured to passively monitor theoutput (audio and/or visual) of the notification generation paths.Hence, the monitoring systems are external and independent of thenotification generation paths. A benefit of this approach is thattypical verification and/or fault detection systems monitor at a pointalong the generation path, wherein faults located downstream of thepoint monitored go undetected.

Feedback provided to the notification generation path (e.g., controller104) may include one or more of verification that the audio/visualsignal was generated, as well as additional parameters associated withthe audio/visual signal (e.g., SNR, decibel level, brightness, etc.)utilized to assess the likelihood that the notification was received bythe operator. In other embodiments, the external/independent monitoringsystem is configured to monitor the response of the operator todetermine whether the audio and/or visual notification was received. Forexample, in some embodiments the visual monitoring system 116 isconfigured to monitor eye movements of the operator to detect whetherthe operator looks at the visual display and/or reacts to the audionotification. In other embodiments, the audio monitoring system 114 isconfigured to monitor auditory responses from the operator acknowledgingreceipt of the audio and/or visual notification.

In some embodiments, in addition to feedback provided by the externalmonitoring systems, input received from vehicle systems (e.g., brakingsystem, steering system) can be utilized to complement feedback receivedfrom the external verification systems to determine whether an operatorreceived a notification. For example, an audio notification indicatingan object located behind a vehicle moving in reverse was likely notaudible to the operator if the brakes are not engaged.

In response to the received feedback, controller 104 may modify one ormore parameters of the audio and/or visual notification and cause thenotification to be generated again. In other embodiments, the controller104 may utilize an alternative generation path. For example, a failedaudio notification may be delivered to the operator using the visualnotification generation path, or vice versa.

FIG. 2 is a block diagram illustrating generation and monitoring ofaudio notifications according to some embodiments. In the embodimentshown in FIG. 2, the audio verification system includes controller 204,audio notification generator 208, amplifier 212, speaker 216, microphone224, and audio signal processor 230. In some embodiments, controller204, audio notification generator 208, amplifier 212, and speaker 216form an audio generation path responsible for generating an audionotification 218 in response to an audio notification request 202. Inthe embodiment shown in FIG. 2, microphone 224 and signal processor 230form an external audio verification system configured to monitor theaudio signals within the interior of the vehicle and to verify thegeneration of audio notifications 218. In some embodiments, microphone224 and/or signal processor 230 may provide additional functionalitybeyond verifying audio notifications 218. For example, microphone 224and signal processor 230 may be utilized as part of a hands-free voicecommand system and/or noise cancelling microphones.

In some embodiments, the audio generation path receives a notificationrequest 202 from one or more systems. In response to the notificationrequest 202, controller 204 generates an audio notification request(hereinafter, “request”) 206 provided to audio notification generator(hereinafter, “generator”) 208. In some embodiments, the request 206 mayidentify the type of audio notification to be generated. For example, adifferent type of audio notification may be generated based on theinformation to be conveyed to the user (e.g., seatbelt notification,blind spot warning, engine warning, etc.). In some embodiments, inaddition to providing a request 206 to generator 208, controller 204provides expected audio notification audio parameters (hereinafter,“parameters”) 234 to audio signal processor 230, which utilizes theparameters 234 to monitor—specifically—for the expected audionotification. In response to the request 206, generator 208 generates anaudio signal 210. In some embodiments, amplifier 212 amplifies the audiosignal 210 and provides the amplified audio signal 214 to speakers 216.In some embodiments, controller 204 provides an input 220 to amplifier212 to control one or more parameters of the amplified signal 214,including amplifier gain (magnitude of the amplified signal 214) and/orthe output speakers 216 that will receive the amplified signal 214. Oneor more speakers 216 convert the amplified audio signal 214 to an audionotification 218. In some embodiments, speakers 216 are dedicated toproviding audio notifications to the operator. In other embodiments,speakers 216 may be multi-purpose, providing audio notifications as wellas operating as entertainment speakers/hand-free calling speakers, etc.In some embodiments, a plurality of speakers 216 may be located atdifferent locations throughout the interior of the cab, whereincontroller 204 may selectively activate one or more of the speakers togenerate the audio notification.

In some embodiments, microphone 224 and audio signal processor 230comprise the external audio verification system. Microphone 224 detectsaudio signals within the interior of the cabin and audio signalprocessor 230 (e.g., digital signal processor (DSP)) analyzes thedetected signal to detect the audio notification 218. In someembodiments, microphone 224 is dedicated for use by the external audioverification system. In some embodiments, microphone 224 is positionedso that sounds sensed by the microphone 224 corresponds closely with thesound heard by the operator. For example, microphone 224 may be locatedin or around the area associated with the operator (e.g., steeringwheel, driver side door, driver seat, etc.). In some embodiments themicrophone 224 utilized by the external audio verification system isalso utilized in other systems, for example hands-free voice commandsystems and/or Bluetooth voice call systems.

Microphone 224 senses all audio sounds within the interior cabin,including both ambient noise 222 and audio notifications 218. As shownin FIG. 2, the recorded audio signal provided by microphone 224 includesrepresentations of the ambient noise 226 (illustrated in FIG. 2 with alighter tone) and the recorded audio notification 228 (illustrated inFIG. 2 with a darker tone). Audio signal analyzer 230 receives thesensed audio signal and analyzes it to detect/verify generation of theaudio notification 218. Audio signal analyzer 230 operates continuouslyto detect/verify generation of one or more audio notification. In thisembodiment, audio signal analyzer 230 may compare the recorded audiosignal to a library of possible audio notifications. In response todetection of one or more of the audio notifications the audio signalanalyzer 230 provides verification feedback to controller 204. In someembodiments, verification feedback provided to controller 204 includesidentification of the notification detected as well as one or moremeasured parameters associated with the audio notification. For example,in some embodiments this may include one or more of magnitude of theaudio notification, magnitude of the audio notification related toambient noise (i.e., signal to noise ratio (SNR)), duration of theaudible chime, intelligibility, formant, tambre, resonance frequencies,attack, decay, etc.

In other embodiments, rather than continuously analyze sensed audiosignals, the audio signal processor 230 analyzes sensed audio signals inresponse to notification parameters 234 received from the controller 204indicating an audio notification request. In some embodiments, theparameters 234 indicates the type of audio notification to be generated.In some embodiments, the parameters 234 may include additionalinformation, such as importance level of the audio notification,amplification setting, verification required, etc. In response, audiosignal processor 230 monitors sensed audio signals for the requestednotification and provides feedback 232 to controller 204 in response tothe detected audio notification 218. In some embodiments, audio signalanalyzer 230 operates to correlate a known signal (i.e., the requestedaudio notification) with the unknown input (i.e., the sensed audiosignal), wherein the correlation is provided in feedback to controller204. Based on the correlation, controller 204 verifies the generation ofthe audio notification 218. Benefits of providing the notificationparameters 234 to the audio signal analyzer 230 include discrete versuscontinuous analysis of the sensed audio signal, which could also helpconserve power. In addition, feedback may include an indication that theparticular audio notification requested by the controller 204 andprovided to audio signal analyzer 230 in the notification parameters 234was not detected. In contrast, if audio signal analyzer 230 iscontinuously monitoring the recorded audio signals for one or more audionotifications without knowledge of whether an audio notification hasbeen requested, signal analyzer 230 cannot provide feedback in responseto a requested notification not being detected (although controller 204may be able to acquiesce from the lack of feedback that no audionotification was detected). In some embodiments, audio signal analyzer230 continuously monitor for audio notifications even if notificationrequests are provided to the audio signal analyzer 230 prior to signalgeneration. In some embodiments, this allows for verification of audionotifications despite errors in generating the notification provided tothe audio signal analyzer 230.

In addition, feedback 232 provided by audio signal analyzer 230 mayinclude one or more parameters measured with respect to the audionotification, such as amplitude, amplitude relative to ambient noise(e.g., SNR), intelligibility, formant, tambre, resonance frequencies,attack, decay, etc. for the audio notification and/or noise. In someembodiments, despite a requested notification being verified by theexternal audio verification system, the one or more measured parametersmay indicate that the audio notification was not audible to the operator(e.g., amplitude and/or SNR below a threshold level). In this way, evenverification that an audio notification was detected may require asecond notification request to correct the defects of the first toensure the audio notification is received/heard by the operator.

In response to feedback received from audio signal analyzer 230,controller 204 may take a variety of different actions. For example, insome embodiments if the feedback 232 verifies generation of the audionotification and measured parameters indicate that the notification wasaudible to the operator, then no further action may be taken. In someembodiments, if the audio notification was detected by the externalaudio verification system but measured parameters such as SNR oramplitude indicate that the audio notification 218 was not audible tothe operator, then controller 204 may generate a second notificationrequest 206 and modify one or more aspects of the audio generation pathsuch as the amplifier gain input 220 to increase the magnitude of theaudio notification 218. In other embodiments, if the audio notificationwas detected by the external audio verification system but measuredparameters such as SNR or amplitude indicated that the audionotification 218 was not audible to the operator, and additionalincreases in volume or speaker selection are not possible or desired orthe notification has a high urgency, then controller 204 may opt togenerate the alert in a different format (e.g., visual notification).Finally, in embodiments in which the audio notification is not detectedby the external audio verification system then controller 204 may makeanother request 206 and increase the amplifier gain 220. In someembodiments, if the audio notification 218 is not detected by theexternal audio verification system and additional increases in volumeare not possible or desired or the notification has a high urgency, thenthe controller 204 may opt to generate the notification in a differentformat (e.g., visual notification). In addition, because the audionotification was not detected at all, controller 204 may generate anerror or alert indicating that the audio generation path—includingcontroller 204, generator 208, amplifier 212, and speaker 216 may beexperiencing a fault that requires service. In the event a fault in theaudio generation path is detected, controller 204 may direct allsubsequent notifications into an alternate notification system (e.g.,visual notification).

In some embodiments, in response to audio notification parameters 234received from controller 204—but prior to generation of the audionotification 218—the external audio verification path may generate anambient noise level feedback 236 provided to controller 204. In someembodiments, microphone 224 senses ambient noise within the cabin andaudio signal analyzer 230 measures the noise level (e.g., amplitude,spectral content, etc.) and provides ambient noise feedback 236 to thecontroller 204. In response to the ambient noise feedback 236,controller 204 may modify delivery of the notification. In someembodiments, controller 204 may determine due to the ambient noisefeedback 236 that an alternative to an audio notification is required,such as a visual notification. In some embodiments, controller 204 maydetermine due to the ambient noise feedback 236 that the amplitude ofthe audio notification will be increased via an increase to theamplifier gain 220 provided to amplifier 212.

In some embodiments, the external audio verification system is utilizedto detect voice commands from the operator regarding receipt of theaudio notification and/or visual notification. For example, in someembodiments, in response to receipt of an audio notification (or visualnotification) the operator provides a verbal/audible verification (e.g.,operator may answer, “Yes”, or “Received”) that is detected by the audiosignal analyzer 230 and provided in feedback to the controller 204.

FIG. 3 is a flowchart illustrating verification of audio notificationsaccording to some embodiment. At least some of the steps included in theflowchart may be optional, as indicated by the dashed lines.

At step 302, an audio notification request is generated. As described inFIG. 2, the request may be generated by controller 204. In someembodiments, at step 304 the independent monitoring system is notifiedof the request. As described with respect to FIG. 2, controller 204 mayprovide the external audio verification system (e.g., audio signalanalyzer 230) with the audio notification parameters 234. For example,if a particular chime is to be generated, the parameters 234 alerts theexternal audio verification system of the particular audio notificationto be detected. In some embodiments, knowledge of the chime to bedetected decreases the processing requirements for analyzing recordedaudio signals.

In some embodiments, at step 306 the independent monitoring system(e.g., microphone 224 and audio signal analyzer 230 shown in FIG. 2)senses ambient sound within the cabin and at step 308 provides feedbackutilized to modify the notification parameters (e.g., increaseamplitude, select speakers, etc.). For example, conditions that mayaffect the ambient noise level includes windows being down, HVAC fans,outside noise, conversation of passengers within the vehicle, personalaudio, etc. In some embodiments, measuring ambient noise levels prior togeneration of the audio notification ensures the first notification isgenerated at an amplitude likely to be audible to the operator. In otherembodiments, pre-measuring ambient noise levels and providing theresults in feedback to the audio notification generation path (e.g.,controller 204 in FIG. 2) is not required, wherein feedback is onlyprovided in response to a generated notification.

At step 310, the audio notification is generated using the generationpath, which as discussed in FIG. 2 may include audio notificationgenerator 208, amplifier 212, and speaker 216. In embodiments in whichpre-measuring ambient noise is provided at step 306 and 308, thengenerating the audio notification may include modifying audio parameters(e.g., amplifier gain, speaker selection, etc.) within the chimegeneration path to ensure the volume level or amplitude of the audionotification is sufficient to be heard by the operator. If pre-measuringambient noise is not utilized, then audio parameters may be selectedbased on pre-determined values, previously utilized audio parameters, orselected based on other factors such as the criticality of thenotification.

At step 312, the external audio verification system monitors/sensessounds generated within the interior of the cabin. In some embodiments,the microphone utilized to sense sounds located in close proximity tothe location of the operator such that the sounds detected by themicrophone are representative of the sounds heard by the operator. Asdiscussed above, in some embodiments it is beneficial if the microphoneutilized for other systems, such as the microphone utilized forhands-free voice commands/Bluetooth calls, etc. can be utilized tomonitor sound within the interior of the cabin. In particular, becausehands-free voice command and/or Bluetooth microphones for voice callsare placed in close proximity to the operator, these devices will alsobe well-positioned to monitor/verify whether audio notifications aredetectable to the operator.

At step 314, audio detected and/or recorded at step 312 (e.g., bymicrophone 224) is analyzed to verify generation of the expected audionotification. In some embodiments, an audio correlation function isutilized to compare the recorded audio with the expected notification,wherein the result of the correlation function provides an indication ofwhether the audio notification was generated. In some embodiments, inaddition to verification of audio notification generation, one or moreparameters associated with the audio notification are measured (e.g.,signal-to-noise ratio (SNR), amplitude of measured signal, etc.).

At step 316, feedback is provided to the generation path (e.g.,controller 204 shown in FIG. 2). In some embodiments, feedback includesverification the audio notification was generated. In other embodiments,feedback includes measured parameters associated with the audionotification including SNR and/or amplitude of the measured signal.

At step 318 a determination is made whether the audio notificationsatisfied notification requirements. For example, in some embodiments ifthe audio notification was verified, then the requirement has beensatisfied and the process ends at step 319. In other embodiments, thedetermination at step 318 determines whether the audio notificationprovided sufficient notification to the operator. In some embodiments,this determination may rely on whether one or more of the measuredparameters (e.g., amplitude, SNR) of the detected audio notificationsatisfies a threshold value. In the event that the audio notification isnot detected, or one or more of the measured parameters (e.g.,amplitude, SNR) does not satisfy a threshold value, then at step 320 adetermination is made whether the notification should be provided via analternative notification path (e.g., visual notification). For example,if the generation path is already operating at a maximum volume (e.g.,maximum amplification), then at step 322 the notification is providedvia an alternative path (e.g., visual notification). A benefit of thisapproach is that the notification is not confined to a single generationpath and may be modified during operation to overcome conditions outsideof the control of the initial generation path. For example, audionotifications—even in an audio generation path that is workingproperly—may not be heard in the event the windows are down. In thisembodiment, an alternative notification path may be utilized to deliverthe notification (e.g., a visual notification).

If an alternative notification path is not utilized at step 320 then atstep 324 the parameters associated with the audio notification aremodified based on the received feedback and a new audio notification isgenerated at step 310. For example, in some embodiments this may includemodifying the amplification of the audio notification and/or speakersutilized to generate the chime, or even generate a different type ofchime designed to get the operator's attention. Based on the newparameters, at step 310 the new audio notification is generated and theprocess continues until the generated notification satisfies theexternal audio verification system and/or an alternative notificationpath is utilized. In other embodiments, additional inputs may beprovided by other systems, such as an external visual notificationverification system utilized to monitor operator reactions (e.g., eyemovement, gaze tracking) in response to an audio notification to detectreceipt by the operator and/or inputs from other vehicle systems (e.g.,braking system activated in response to rear object alert while backingup). In this way, external systems (i.e., external to the notificationgeneration path) may be utilized to verify generation and receipt ofaudio notifications or chimes.

FIG. 4 is a forward-facing view of an interior 400 of a vehicle thatincludes a display 402 and camera 404. As described in more detail withrespect to FIGS. 5 and 6 below, a visual notification generation pathincludes a plurality of components configured to generate visualnotifications in one or more locations, such as display 402. In someembodiments, the camera 404 is part of an external visual notificationverification system configured to monitor one or more of the display 402and/or the operator (not shown) of the vehicle in order to verifygeneration of a visual notification and/or receipt of the visualnotification by the operator. For example, in some embodiments, thecamera 404 captures images/videos of the display 402 and providesfeedback to the visual notification generation path verifying whetherthe visual notification was displayed properly. In some embodiments,verification may also include analyzing the captured images/video toderive one or more parameters associated with the visual notification(e.g., brightness, contrast, missing pixels, etc.) associated with thedisplayed visual notification. In some embodiments, camera 404 capturesimages/video of the operator and provides feedback to the visualnotification generation path verifying that the operator noticed thevisual notification. In some embodiments, this may include analyzing eyemovements/gaze of the operator to determine whether the operator lookedat the visual notification. In some embodiments, analysis of the displayand of the operator are utilized in conjunction with one another toverify generation and/or receipt of the visual notification.

Referring to FIG. 5, a block diagram of the visual notificationgeneration path, including controller 502, system-on-chip (SoC) 506,serializer/deserializer 510, and display 514 is illustrated, along witha block diagram of the external visual notification verification system,which includes one or more cameras 516 and image analyzer 520.

Similar to that discussed in FIG. 2 with respect to the chime generationpath, the visual notification generation path includes a controller 502configured to receive visual notification requests 501. In response tothe visual notification request, controller 502 generates a visualrequest that is provided to the SoC 506. In some embodiments, controller502 may also generate an expected visual notification 524 provided tothe external visual notification verification system—specifically toimage analyzer 520—alerting the visual monitoring system of the visualnotification to be generated. SoC 506 processes the request for aparticular visual notification to be displayed and acts to add thevisual notification to the overall display. In some embodiments, if thevisual notification is simply lighting a lamp/LED on the dash (e.g.,check engine light), then SoC 506 may simply provide the instruction tolight the particular lamp/LED. However, in many cases the display 514 isa digital screen capable of displaying images. In some embodiments, SoC506 may incorporate a plurality of images—including visualnotifications—into a composite image to be displayed by the display 514.Based on the output 508 provided by the SoC 506, serializer/deserializer510 converts the composite image to the input required by display 514.

In some embodiments, camera 516 is oriented to monitor the display 514.In other embodiments, the video signal 512 is bifurcated into first andsecond video signals. The first video signal 512 is sent to the displayand the second video signal 515 is provided as an input to camera 516(or some other device capable of receiving a video input). In someembodiments, relying solely on the second video signal 515 forverification of visual notifications will not capture faults residing onthe display 514 (e.g., missing pixels, poor brightness, etc.) that mayresult in the visual notification—even if correctly generated by the SoC506 and serializer/deserializer 510—failing to communicate informationto the operator. In some embodiments, both videos/images captured bycamera 516 and the second video input 515 provided to camera 516 may beanalyzed to verify generation of the visual notification and/or todetermine the source of the fault if one is detected.

Monitored images/video 518—either captured by camera 516 and/or receivedvia second video input 515—are provided to image analyzer 520. In someembodiments, image analyzer 520 analyzes all received images fordetection of one or more visual notifications. For example, imageanalyzer 520 may maintain a library of visual notifications and maycontinuously monitor the recorded image/video 518 to determine whetherany of the library of visual notifications is detected within therecorded images/video 518. In response to a detected visualnotification, image analyzer 520 provides feedback 522 to the controller502 identifying the visual notification detected. In other embodiments,in response to an expected visual notification 524, image analyzer 520analyzes captured images/video 518 to verify whether the visualnotification was properly displayed by display 514. In some embodiments,verifying a visual notification in response to an expected visualnotification communication alleviates requirements that the imageanalyzer 520 continuously analyze captured images/video 518 to detectvisual notifications. A variety of algorithms may be utilized to detectvisual notifications. In some embodiments, image analyzer 520 utilizes avisual pattern matching algorithm via neural networks to identify knownvisual notifications. In other embodiments, image analyzer 520 mayutilize a digital signal processor (DSP) image correlation algorithm.

In some embodiments, in addition to verifying generation of a visualnotification, feedback 522 may include one or more parameters associatedwith the visual notification. For example, in some embodiments imageanalyzer 520 may also provide feedback regarding brightness of thevisual notification, brightness as compared with the ambient lightingconditions, contrast of the visual display as compared with thebackground of the display, bad graphics panel (e.g., dead pixels), etc.

In some embodiments, either in conjunction with monitoring of thedisplay or in lieu of monitoring the display, camera 516 may be orientedto monitor the operator of the vehicle to detect whether a visualnotification was received by the operator. In some embodiments,controller 502 communicates an expected visual notification 524 to theexternal visual notification system. In response, image analyzer 520begins analyzing images and/or video 518 provided by camera 516 toverify operator reaction to the provided visual notification. Forexample, in some embodiments image analyzer 520 utilizes gaze orattention tracking to verify that the operator looked in the directionof the generated visual notification for a sufficiently long period oftime. In some embodiments, image analyzer 520 utilizes gaze or attentiontracking to verify that the operator responded to the notificationprovided by looking in the direction required by the alert. For example,if the visual notification is a rear object detected alert, then theoperator's gaze should be directed to a rearview mirror and/or backupcamera display for a sufficient period of time. In some embodiments,either alone or in combination with gaze or attention tracking, imageanalyzer 520 may monitor visual biometric parameters associated with theoperator to detect a change in status in response to the visualnotification. For example, in some embodiments image analyzer 520 maymonitor eye reaction or facial movements to determine whether theoperator received the visual notification. In other embodiments,additional sensors (not shown) may be utilized to collect biometric dataassociated with the operator. Biometric data may include heart rate,blood pressure, and/or movement of the operator. These biometricparameters may be analyzed alone or in combination with video/image toverify operator receipt of the visual notification. The result of theanalysis is provided to the visual notification generation path (e.g.,controller 502) and may be utilized in conjunction with other feedbackto determine whether the visual notification was properly generatedand/or received by the operator.

Based on the received feedback, controller 502 may take steps tomitigate a failure to generate the visual notification or failure togenerate a visual notification likely to be seen by the operator. Thesteps taken by controller 502 may depend on whether the visualnotification was verified but monitored parameters indicate that it waslikely not received or detected by the operator, or whether the visualnotification generation path failed to generate the visual notificationentirely. For example, if the visual notification was not generated,that indicates a vault in the visual notification generation path. Insome embodiments, controller 502 may react by deciding to deliver thenotification via an alternative notification path (e.g., audiogeneration path). If the visual notification was generated, but one ormore parameters indicate that it is likely that the operator did notreceive the notification, then the controller 502 may request generationof a second visual notification wherein one or more parameters of thevisual notification are modified to improve the likelihood the visualnotification is received. This may include modifying the location atwhich the visual notification is generated, modifying the brightness ofthe backlighting, and/or contrast between the visual notification andthe background, changing thebackground/brightness/coloring/opacity/animation/placement or otheraspects of the visual display notification itself, or sending adifferent visual notification.

In some embodiments, the independent external visual monitoring systemmay also be utilized to monitor ambient conditions within the cabin andprovide the controller 502 with feedback regarding the ambientconditions prior to generation of the visual notification. Controller502 may utilize the monitored ambient conditions to modify one or moreparameters associated with the visual notification being generated.Parameters monitored to determine ambient conditions may includebrightness (e.g., daytime, nighttime), ratio of brightness of the screenas compared with the interior of cabin, obscuration warnings withrespect to the display, and/or other parameters associated withconditions within the cabin and/or conditions associated with thedisplay. Based on these parameters, controller 502 may modify the visualnotification generated.

FIG. 6 is a flowchart illustrating verification of visual displaynotifications according to some embodiment. In the embodiment shown inFIG. 6, at least some of the boxes indicating steps performed are shownwith dashed lines to indicate that these steps may be optionallyperformed in some embodiments.

At step 602, a display notification request is generated. As describedin FIG. 5, the display notification request may be generated bycontroller 502. In some embodiments, at step 604 the independent visualmonitoring system is notified of the visual display notificationrequest. As described with respect to FIG. 5, controller 502 may providethe independent monitoring system (e.g., image analyzer 520) with thedisplay notification request, which may include identification of thevisual notification to be generated.

In some embodiments, at step 606 the independent visual monitoringsystem (e.g., camera 516 and image analyzer 520 shown in FIG. 5) recordsimage and/or video data from within the cabin and utilizes the recordeddata to determine ambient visual conditions within the cabin (e.g.,brightness, obscuration of display, etc.). Ambient visual conditions areprovided in feedback to the visual notification path (e.g., controller502 shown in FIG. 5). At step 608 the controller 502 may selectivelymodify one or more aspects of the visual notification generated based onthe ambient visual conditions feedback received. For example, this mayinclude modifying backlighting associated with the display (e.g.,increased in response to bright ambient conditions), moving the locationof the visual notification to an unobscured display surface, modifyingthe background/brightness/coloring/opacity/animation/placement or otheraspects of the visual display notification itself, or other steps toincrease the likelihood that the visual notification is seen by theoperator. The purpose of providing initial feedback to the visualnotification generation path is to increase the likelihood that thefirst visual notification is generated and received (i.e., noticed) bythe operator, without having to generate a second visual notification tomitigate the failure of the first visual notification to be seen by theoperator.

At step 610, the visual notification is generated using the visualnotification generation path, which as discussed in FIG. 5 may include aSoC 506, serializer/deserializer 510, and/or display 514. In embodimentsin which pre-measuring ambient conditions is provided at steps 606 and608, then generating the visual notification may include modifying oneor more parameters associated with the visual notification generationpath (e.g., display backlighting brightness, notification location,notification aspects, etc.) to ensure delivery of the notification tothe operator.

At step 612, the independent visual monitoring path records imagesand/or video data to verify delivery of the visual notification. In someembodiments, this includes monitoring images/video data focused on thedisplay delivering the visual notifications. In other embodiments thismay include monitoring image/video data focused on the operator of thevehicle to determine whether the operator noticed/received the visualnotification. In some embodiments, one or more cameras may be utilizedto monitor multiple display locations and/or operator response.

At step 614, image/video data monitored at step 612 (e.g., by camera516, shown in FIG. 5) is analyzed to verify generation of the expectedvisual notification and/or receipt of the visual notification by theoperator. In some embodiments, a plurality of algorithms may be utilizedto recognize/correlate an expected visual notification with monitoredimage/video data, including utilization of image recognition usingneural networks, image correlation using DSP, and/or image recognitionusing artificial intelligence. For example, a neural network utilizestraining data to identify visual notifications presented on a display.In this way, a particular visual notification can be identified within acomposite image. In some embodiments, the output provided by therecognition algorithm indicates whether the visual notification has beendetected. This may be a binary yes/no output, or it may include anumerical confidence level associated with the determination. In someembodiments, in addition to a verification of whether the visualnotification was properly generated, one or more parameters is measuredthat corresponds with the visual notification. For example, in someembodiments this may include monitoring brightness of the visualnotification generation, contrast of the visual notification as comparedwith the background, identification of faults (e.g., missing pixels) inthe visual notification, and/or other parameters associated with thevisual notification.

In other embodiments, image/video data monitors the operator's reactionto a visual notification and is analyzed at step 614 to determinewhether the operator was made aware of the visual notification. In thistype of analysis, the visual notification itself is not detected; ratherthe operator's reaction to the visual notification is monitored todetermine whether the operator received the notification. In someembodiments, this may include gaze analysis to determine whether theoperator has looked at the region in which the visual notification wasgenerated for a sufficiently long period of time. In some embodiments,gaze analysis may be utilized to determine if the operator looked in thedirection indicated by the alert. For example, a visual notificationindication that a rear object has been detected while the car is backingup should result in the operator looking at the rear-view mirror and/orthe display for the rear camera. If the driver does not look at eitherof these, this is an indication that the visual notification was notreceived by the operator.

At step 616, the results of the analysis is provided in feedback to thevisual notification generation path (e.g., controller 502 shown in FIG.5). In some embodiments, feedback includes verification the visualnotification was generated. In other embodiments, feedback includesmeasured parameters associated with the visual notification (e.g.,brightness, contrast with background, obscuring of the visualnotification, missing pixels, etc.). In other embodiments, feedback mayinclude verification that the operator looked at the visualnotification.

At step 618, a determination is made whether the visual notificationgenerated satisfied the notification requirements. For example, in someembodiments if the visual notification was verified, then therequirement has been satisfied and the process ends at step 619. Inother embodiments, the determination at step 618 determines whether thevisual notification provided sufficient notification to the operator. Insome embodiments, this determination may additionally be based onmeasured parameters associated with the visual notification (e.g.,brightness, contrast with the background, obscuring of the visualnotification, missing pixels, etc.). For example, the independent visualmonitoring system may detect the presence of the visual notification,but due to an object obscuring at least a portion of the visualnotification it may be determined that the visual notification wasinsufficiently visible to provide notice to the operator. Similarly, theindependent visual monitoring system may detect the presence of thevisual notification, but analysis of the operator may determine that theoperator did not look in the direction of the visual notification andtherefore likely did not receive the notification. In some embodiments,in addition to inputs received from the independent visual monitoringsystem with respect to either verification of the visual notificationand/or verification that the operator looked at the visual notification,in some embodiments the actions of the operator may also be utilized todetermine whether the visual notification was received by the operator.For example, if the visual notification indicates that an object wasdetected behind the vehicle while the vehicle is backing up and theoperator has not engaged the brakes, this indicates that the operatordid not receive the visual notification. Each of these feedbacks may beutilized in combination with one another to determine whether the visualnotification was properly generated and/or received by the operator.

If, at step 618 it is determined that the visual notification was notgenerated, or one or more of the measured parameters or external inputsindicates that the visual notification was not received, then at step620 a determination is made whether the notification should be providedvia an alternative notification path (e.g., audio notification). Forexample, if the brightness of the visual display is already operating ata maximum brightness, or if no alternative display is available fordisplaying the visual notification, then at step 622 the notification isprovided via an alternative path (e.g., audio notification). A benefitof this approach is that the notification is not confined to a singlegeneration path and may be modified during operation to overcomeconditions outside of the control of the initial generation path. Forexample, visual notifications—even if the visual generation path isworking properly—may not be visible as a result of the operatorobscuring the display with an object. In this embodiment, an alternativenotification path may be utilized to deliver the notification (e.g., anaudio notification).

If an alternative notification path is not utilized at step 620 then atstep 624 one or more parameters of the visual notification are modifiedbased on the received feedback and a new visual notification isgenerated at step 610. For example, in some embodiments visualnotification parameters modified include the backlighting/brightness ofthe visual notification. In other embodiments, the visual notificationmay be generated at an alternative display or screen. In still otherembodiments, the visual notification itself may be changed such as itsbackground/brightness/coloring/opacity/animation/placement or otheraspects. Based on the modified parameters/instructions, at step 610 thenew visual notification is generated, and the process continues untilthe generated notification satisfies the independent visual monitoringsystem and/or an alternative notification path is utilized.

Discussion of Possible Embodiments

The following are non-exclusive descriptions of possible embodiments ofthe present invention.

According to some aspects, a method of verifying audio notificationsgenerated within a vehicle is provided. The method may includegenerating an audio notification using an audio generation path thatincludes at least a controller, an audio notification generator and aspeaker. The controller may generate an audio notification requestutilized by the audio notification generator to generate an audiosignal, wherein the speaker may then convert the audio signal into theaudio notification. The method may further include monitoring soundswithin the vehicle using a microphone separate from the audio generationpath and analyzing the sounds monitored by the microphone using an audioanalyzer, wherein the audio analyzer verifies whether the audionotification was generated. Feedback may be provided to the audiogeneration path regarding whether generation of the audio notificationwas verified.

The method of the preceding paragraph can optionally include,additionally and/or alternatively any, one or more of the followingfeatures, configurations and/or additional components.

For example, analyzing the sounds within the vehicle may includemeasuring one or more parameters of the audio notification.

In some embodiments, the one or more parameters of the audionotification measured include an amplitude of ambient noise and asignal-to-noise ratio (SNR) of the audio notification.

In some embodiments, providing feedback to the audio generation path mayfurther include providing one or more of the measured parameters infeedback to the audio generation path.

In some embodiments, the method may include amplifying the audio signalgenerated by the audio notification generator using an amplifierincluded as part of the audio generation path, wherein the speakerconverts the amplified audio signal to the audio notification.

In some embodiments, the controller may selectively increase/decreasethe amplification provided by the amplifier based on feedback providedto the generation path regarding verification of the audio notification.

In some embodiments, the method may include providing a notification tothe audio analyzer of the audio notification request generated by thecontroller, wherein the audio analyzer analyzes sounds monitored by themicrophone for a period of time following the audio notificationrequest.

In some embodiments, the method may further include measuring ambientsound levels using the microphone and audio analyzer prior to generationof an audio notification, providing feedback to the controller regardingthe measured ambient noise levels, and selectively increasing/decreasingthe amplification provided by the amplifier based on the measuredambient noise levels.

In some embodiments, the method may further include generating analternative notification in response to feedback indicating failure togenerate the audio notification.

In some embodiments, the alternative notification may be an audionotification generated at an alternative speaker.

In some embodiments, the alternative notification may be a visualindicator displayed to an operator.

According to another aspect, an audio notification system may include anaudio notification generation path and an external audio verificationsystem. The notification generation path may include at least acontroller, an audio notification generator, and a speaker, wherein thecontroller may generate an audio notification request and the audiogenerator may in response generate an audio notification signal. Thespeaker may then convert the audio notification signal into an audiooutput provided as an audio notification to an operator of the vehicle.The external audio verification system may include at least a microphoneand an audio analyzer, wherein the microphone may monitor audio signalswithin an interior cabin of the vehicle and the audio analyzer mayanalyze monitored audio signals to verify whether the audio notificationrequested by the controller was properly generated. Verification may beprovided as feedback to the audio notification generation path.

The audio notification system of the preceding paragraph can optionallyinclude, additionally and/or alternatively any, one or more of thefollowing features, configurations and/or additional components.

For example, the microphone may be utilized to provide hands freecalling and voice recognition.

In some embodiments, the audio analyzer may correlate the monitoredaudio signals with one or more possible audio notifications to verifygeneration of the audio notification.

In some embodiments, the controller may provide an expected audionotification signal identifying an audio notification to be generated tothe external audio verification system to notify the audio analyzer tobegin analyzing the monitored audio signals for the identified audionotification.

In some embodiments, the microphone and the audio analyzer may monitorambient noise levels and provide an ambient noise level feedback to theaudio notification generation path.

In some embodiments, the audio notification generation path may utilizethe ambient noise levels to selectively modify a volume of the audionotification generated.

In some embodiments the audio notification system may further include avisual display configured to display visual notifications, wherein theaudio notification generation path may generate a visual notification inresponse to feedback indicating a failure to generate the audionotification.

According to another aspect, a method of verifying audio notificationsgenerated within a vehicle may include receiving feedback from anexternal audio monitoring system that includes a microphone and an audiosignal analyzer regarding ambient sound levels measured within aninterior cabin of a vehicle. An audio notification may be generatedusing an audio generation path that may includes at least a controller,an audio notification generator, an amplifier and a speaker, wherein thecontroller may generate an audio notification request utilized by theaudio notification generator to generate an audio notification signalbased on the ambient sound levels measured by the external audiomonitoring system. The speaker may convert the audio notification signalinto the audio notification. The method may further include monitoringsounds within the vehicle using a microphone separate from the audiogeneration path and analyzing the sounds monitored by the microphoneusing an audio analyzer, wherein the audio analyzer verifies whether theaudio notification was generated. Feedback may be provided to the audiogeneration path regarding whether generation of the audio notificationwas verified.

The method of the preceding paragraph can optionally include,additionally and/or alternatively any, one or more of the followingfeatures, configurations and/or additional components.

For example, the method may further include generating a visualnotification using a visual generation path in response to feedbackindicating that the audio notification was not verified.

1. A method of verifying notifications generated within a vehicle, themethod comprising: generating an output using a generation path thatincludes at least a controller and an output device; and verifyinggeneration of the output with a verification path is separate from thegeneration path.
 2. The method of claim 1, wherein the generation pathis a visual generation path that generates a visual notification on adisplay.
 3. The method of claim 2, further including: monitoring aninterior of a vehicle with a camera, wherein verifying generation of theoutput includes utilizing an image analyzer to analyze images capturedby the camera to verify generation of the visual notification on thedisplay.
 4. The method of claim 3, wherein monitoring an interior of thevehicle with a camera includes monitoring the display included as partof the generation path.
 5. The method of claim 4, wherein the imageanalyzer compares images captured by the camera to a library of visualnotifications to detect the presence of one or more visual notificationsincluded in the library, wherein in response to detecting one or morethe visual notifications a verification output is provided to thegeneration path verifying generation of the visual output.
 6. The methodof claim 4, wherein the generation path provides the verification pathwith an expected visual notification indicating the visual notificationto be generated, wherein the image analyzer analyzes captured images forthe visual notification indicated to be generated.
 7. The method ofclaim 4, wherein the image analyzer measures and provides feedback tothe visual notification path regarding one or more of brightness of thevisual notification, brightness of the visual notification as comparedwith ambient lighting, contrast of the visual notification and/ordisplay as compared with a background brightness of the display, andfault graphics panel or pixels.
 8. The method of claim 3, whereinmonitoring the interior of the vehicle with a camera includes monitoringfacial features of an occupant of the vehicle.
 9. The method of claim 8,wherein utilizing an image analyzer to analyzer images captured by thecamera includes monitoring one or more of eye reaction and facialmovements to determine whether the occupant of the vehicle received thevisual notification.
 10. The method of claim 1, further including:generating an alternative output using an alternative generation path inresponse to the verification path indicating that the output was notproperly generated.
 11. The method of claim 10, wherein the generationpath is a visual generation path and the alternative generation path isan audio generation path.
 12. A visual notification system for use invehicles, the system comprising: a visual notification generation paththat includes at least a controller and a display for displaying visualnotifications requested by the controller; and a visual verificationpath external to the visual notification generation path that includesat least a camera and an image analyzer, wherein the camera monitors aninterior of the vehicle and the image analyzer analyzes images capturedby the camera and generates a verification output provided to the visualnotification generation path verifying whether the visual notificationwas properly generated by the visual notification generation path. 13.The visual notification system of claim 12, wherein the camera capturesimages corresponding with the display and wherein the image analyzeranalyzes images captured by the camera to verify generation of thevisual notification on the display.
 14. The visual notification systemof claim 13, wherein the image analyzer compares images captured by thecamera to a library of visual notifications to detect the presence ofone or more visual notifications included in the library.
 15. The visualnotification system of claim 13, wherein the visual notificationgeneration path provides the visual verification path with an expectedvisual notification indicating the visual notification to be generated,wherein the image analyzer analyzes captured images for the visualnotification indicated to be generated.
 16. The visual notificationsystem of claim 13, wherein the image analyzer measures and providesfeedback to the visual notification path regarding one or more ofbrightness of the visual notification, brightness of the visualnotification as compared with ambient lighting, contrast of the visualnotification and/or display as compared with a background brightness ofthe display, and fault graphics panel or pixels.
 17. The visualnotification system of claim 12, wherein the camera is oriented tomonitor facial features of an occupant of the vehicle and wherein theimage analyzer monitors one or more of eye reaction and facial movementsto determine whether the occupant of the vehicle received the visualnotification.
 18. The visual notification system of claim 12, whereinthe visual verification path includes a system on chip (SoC) thatprocesses visual notification requests received from the controller togenerate a composite image for display that includes the visualnotification requested.
 19. The visual notification system of claim 12,further including: an audio notification generation path that includesat least a controller and a speaker for generating audio notificationsrequested by the controller, wherein the audio notification generationpath is utilized as an alternative generation path in response to theverification output indicating that the visual notification was notproperly displayed or received by an operator.
 20. The visualnotification system of claim 19, further including: an audioverification path external to the audio notification generation paththat includes at least a microphone and an audio signal analyzer,wherein the microphone monitors sounds within an interior of the vehicleand the audio signal analyzer analyzes captured sounds and generates anaudio verification output provided to the audio notification generationpath verifying whether the audio notification was properly generated bythe audio notification generation path.